#KV cache

4 articles

Tech Apr 2, 2026 11 min

SwiftLM is a Swift-based LLM inference server that integrates TurboQuant and SSD streaming into Metal shaders

SwiftLM, an Apple Silicon–only MLX inference server, provides a native Metal implementation of TurboQuant V2+V3 hybrid KV‑cache compression and NVMe SSD expert streaming.

Apple Silicon LLM MLX Local LLM Inference optimization KV cache MoE Swift

Tech Mar 31, 2026 8 min

Scaling Qwen3.5-35B-A3B from 4K to 65K Context with Only 800MB Extra VRAM

Qwen3.5-35B-A3B is an SSM+Attention hybrid where only 10 of 40 layers use KV cache. Expanding ctx-size from 4096 to 65536 on llama-server added just 800MB VRAM with zero speed loss. Includes q8_0 KV quantization benchmarks and TurboQuant status.

LLM Local LLM llama.cpp AMD Vulkan KV Cache Qwen Benchmark

Tech Mar 25, 2026 17 min

Hypura’s NVMe Streaming and TurboQuant’s KV Cache Quantization

Hypura breaks away from llama.cpp’s mmap design and streams even dense models with a three-tier NVMe placement, while TurboQuant eliminates quantization-constant overhead via a polar-coordinate transform. Includes a design comparison with Flash‑MoE and a review of scenarios where KV‑cache compression actually helps.

LLM Local LLM Quantization Apple Silicon Inference Optimization KV Cache Rust

Tech Feb 20, 2026 updated 13 min

Accelerating LLM Inference: CDLM and Attention Matching KV Compaction

Two February 2026 papers on reducing inference cost: Together AI’s Consistency DLM (up to 14.5× faster) and MIT/Harvard’s Attention Matching KV compaction (50× compaction in seconds).

AI LLM Inference optimization KV cache Diffusion models